Device including plurality of markers

ABSTRACT

An input device includes a case body and a plurality of markers  30   d,    30   e , and  30   p  configured to emit light to the outside of the case body. When an angle difference between the normal directions of the two markers  30   d  and  30   e  is less than a predetermined threshold value, a distance between the two markers  30   d  and  30   e  is equal to or greater than a predetermined first distance. When an angle difference between the normal directions of the two markers  30   d  and  30   p  is equal to or greater than a predetermined threshold value, a distance between the two markers  30   d  and  30   p  is equal to or greater than a predetermined second distance, and the second distance is shorter than the first distance.

TECHNICAL FIELD

The present invention relates to a device including a plurality ofmarkers.

BACKGROUND ART

PTL 1 discloses a game device that acquires a frame image obtained byimaging a space in front of the game device, that estimates, from theposition of a light emitting diode (LED) image of a game controller inthe frame image, position information and posture information regardingthe game controller in a real space, and that reflects the estimatedposition information and/or posture information on the processing of agame application.

CITATION LIST Patent Literature [PTL 1]

Japanese Patent Laid-open No. 2007-296248

SUMMARY Technical Problem

A head-mounted display (HMD) is fitted to the head of a user to providethe user with a virtual reality (VR) video space. The user who wears theHMD operates operation buttons in an input device and can therebyproduce various inputs to the video space.

In recent years, techniques for tracking the position and posture of adevice to reflect information obtained, on a three-dimensional (3D)model in a VR space, have widely been used. The movement of a playercharacter or a game object in a game space is synchronized with changesin the position and posture of a device to be tracked, and thus anintuitive operation by a user is realized. A plurality of markers suchas light emitting elements are used for the tracking of the device, animage obtained by imaging the plurality of markers is analyzed, and thepositions of marker images within the image are thus identified, withthe result that the position and posture of the device in a real spaceare estimated. In order to track the position and posture of the devicewith a high degree of accuracy, it is necessary to accurately identifythe positions of the marker images within the image obtained by imagingthe device.

Hence, an object of the present invention is to provide a device inwhich a plurality of markers are arranged to realize highly accuratetracking. Note that, while the device may be an input device thatincludes operation buttons, the device may be a device that is simply atarget to be tracked including no operation members.

Solution to Problem

In order to solve the problem described above, a device according to anaspect of the present invention includes a case body and a plurality ofmarkers configured to emit light to the outside of the case body, andwhen an angle difference between the normal directions of two markers isless than a predetermined threshold value, a distance between the twomarkers is equal to or greater than a predetermined distance.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram depicting an example of the configuration of aninformation processing system in an embodiment.

FIG. 2 is a diagram depicting an example of the external shape of anHMD.

FIG. 3 is a diagram depicting a functional block of the HMD.

FIG. 4 depicts diagrams illustrating the external shape of an inputdevice.

FIG. 5 is a diagram depicting a functional block of the input device.

FIG. 6 is a diagram depicting an example of part of an image obtained byimaging the input device.

FIG. 7 is a diagram depicting the shape of a right side surface of theinput device.

FIG. 8 is a diagram depicting a schematic configuration of a curvedportion.

FIG. 9 is a diagram for illustrating the normal directions of twomarkers on the same surface.

DESCRIPTION OF EMBODIMENT

FIG. 1 depicts an example of the configuration of an informationprocessing system 1 in an embodiment. The information processing system1 includes an information processing device 10, a recording device 11,an HMD 100, an input device 16 that is operated with fingers of a user,and an output device 15 that outputs images and sounds. The outputdevice 15 may be a television set. The information processing device 10is connected through an access point (AP) 17 to an external network 2such as the Internet. The AP 17 has the functions of a wireless accesspoint and a router, and the information processing device 10 may beconnected to the AP 17 with a cable or may be connected thereto with aknown wireless communication protocol.

The recording device 11 records system software and applications such asgame software. The information processing device 10 may download thegame software to the recording device 11 from a content server throughthe network 2. The information processing device 10 executes the gamesoftware and supplies image data and sound data of the game to the HMD100. The information processing device 10 and the HMD 100 may beconnected to each other with a known wireless communication protocol ormay be connected to each other with a cable.

The HMD 100 is a display device that is fitted to the head by the user,to display images on display panels located in front of the eyes. TheHMD 100 individually displays an image for the left eye on a displaypanel for the left eye and an image for the right eye on a display panelfor the right eye. These images form parallax images seen from left andright eyepoints to realize a stereoscopic view. In order for the user tolook at the display panels through optical lenses, the informationprocessing device 10 supplies, to the HMD 100, parallax image dataobtained by correcting optical distortion caused by the lenses.

Although the output device 15 is not needed for the user who wears theHMD 100, preparing the output device 15 enables another user to see thedisplay image of the output device 15. While the information processingdevice 10 may display, on the output device 15, the same image as theimage seen by the user who is wearing the HMD 100, another image may bedisplayed. For example, when the user who is wearing the HMD 100 plays agame together with another user, a game image from the eyepoint of acharacter of the other user may be displayed from the output device 15.

The information processing device 10 and the input device 16 may beconnected to each other with a known wireless communication protocol ormay be connected to each other with a cable. The input device 16includes a plurality of operation members such as operation buttons, andthe user operates the operation members with the fingers while graspingthe input device 16. When the information processing device 10 executesa game, the input device 16 is used as a game controller. The inputdevice 16 is provided with a posture sensor including a three-axisacceleration sensor and a three-axis gyro sensor, and transmits sensordata to the information processing device 10 at a predetermined cycle(for example, 1600 Hz).

The game of the embodiment handles not only the operation information ofthe operation members in the input device 16 but also the operationinformation of the position, the posture, the movement, and the like ofthe input device 16, and reflects the operation information on themovement of the player character within a virtual three-dimensionalspace. For example, the operation information of the operation membersmay be used as information for moving the player character, and theoperation information of the position, the posture, the movement and thelike of the input device 16 may be used as information for moving an armof the player character. For example, in a battle scene within the game,the movement of the input device 16 is reflected on the movement of theplayer character holding a weapon, and thus an intuitive operation bythe user is realized, with the result that the immersion of the user inthe game is enhanced.

In order to track the position and posture of the input device 16, aplurality of markers (light emitting portions) that can be imaged withimage sensing devices 14 mounted on the HMD 100 are provided on theinput device 16. The information processing device 10 analyzes an imageobtained by imaging the input device 16, to estimate the positioninformation and posture information of the input device 16 in a realspace, and provides the estimated position information and postureinformation to the game.

On the HMD 100, a plurality of image sensing devices 14 are mounted. Theplurality of image sensing devices 14 are attached to differentpositions of the front surface of the HMD 100 with different posturessuch that the overall imaging range obtained by adding up the imagingranges of the image sensing devices 14 includes the entire field of viewof the user. The image sensing device 14 is preferably an image sensorthat can acquire the images of the plurality of markers in the inputdevice 16. For example, when the marker emits visible light, the imagesensing device 14 includes a visible light sensor, such as a chargecoupled device (CCD) sensor or a complementary metal oxide semiconductor(CMOS) sensor, that is used in a general digital video camera. When themarker emits invisible light, the image sensing device 14 includes aninvisible light sensor. The plurality of image sensing devices 14 imagea space in front of the user at a synchronized timing and predeterminedintervals (for example, 60 frames/second), and transmit image dataobtained by imaging the input device 16 to the information processingdevice 10.

The information processing device 10 identifies the positions of aplurality of marker images of the input device 16 included in a capturedimage. Note that, while the one input device 16 may be imaged with aplurality of image sensing devices 14 at the same timing, since theattachment positions and attachment postures of the image sensingdevices 14 are known, in such a case, the information processing device10 combines a plurality of captured images to identify the positions ofthe marker images.

The three-dimensional shape of the input device 16 and the positioncoordinates of the plurality of markers arranged on the surface thereofare known, and thus, the information processing device 10 estimates theposition coordinates and the posture of the input device 16, based onthe distribution of the marker images within the captured image. Theposition coordinates of the input device 16 may be position coordinatesin a three-dimensional space with a reference position being an origin,and the reference position may be position coordinates (longitude andlatitude) set before the start of the game.

Note that the information processing device 10 can also estimate theposition coordinates and the posture of the input device 16 by using thesensor data detected with the posture sensor in the input device 16.Hence, the information processing device 10 of the embodiment uses theresult of the estimation based on the captured image obtained with theimage sensing devices 14 and the result of the estimation based on thesensor data, so as to perform tracking processing on the input device 16with a high degree of accuracy.

FIG. 2 depicts an example of the external shape of the HMD 100. The HMD100 includes an output mechanism portion 102 and a fitting mechanismportion 104. The fitting mechanism portion 104 includes a fitting band106 that is put on by the user to be worn around the head so as to fixthe HMD 100 to the head. The fitting band 106 has a material or astructure in which the length thereof can be adjusted according to thehead circumference of the user.

The output mechanism portion 102 includes a housing 108 having a shapethat covers the left and right eyes in a state where the user is wearingthe HMD 100, and also includes, therewithin, the display panels directlyopposite the eyes when the HMD 100 is worn. The display panel may be aliquid crystal panel, an organic electroluminescence (EL) panel, or thelike. Within the housing 108, a pair of left and right optical lensesthat are located between the display panels and the eyes of the user toenlarge the viewing angle of the user are further included. The HMD 100may further include speakers and earphones in positions corresponding tothe ears of the user or may be configured such that external headphonesare connected thereto.

A plurality of image sensing devices 14 a, 14 b, 14 c, and 14 d areprovided on an outer surface on the forward side of the housing 108.With reference to the direction of the line of sight of the user, theimage sensing device 14 a is attached to an upper right corner of theouter surface on the forward side such that a camera optical axis isdirected diagonally upward to the right, the image sensing device 14 bis attached to an upper left corner of the outer surface on the forwardside such that the camera optical axis is directed diagonally upward tothe left, the image sensing device 14 c is attached to a lower rightcorner of the outer surface on the forward side such that the cameraoptical axis is directed diagonally downward to the right, and the imagesensing device 14 d is attached to a lower left corner of the outersurface on the forward side such that the camera optical axis isdirected diagonally downward to the left. A plurality of image sensingdevices 14 are installed in this way, and thus, the overall imagingrange obtained by adding up the imaging ranges of the image sensingdevices 14 includes the entire field of view of the user. The field ofview of the user described above may be the field of view of the user ina virtual three-dimensional space.

The HMD 100 transmits, to the information processing device 10, thesensor data detected with the posture sensor and the image data obtainedby imaging by the image sensing devices 14, and receives game image dataand game sound data generated in the information processing device 10.

FIG. 3 depicts a functional block of the HMD 100. A control portion 120is a main processor that processes and outputs various types of datasuch as the image data, the sound data, and the sensor data andcommands. A storage portion 122 temporarily stores the data, thecommands, and the like processed by the control portion 120. The posturesensor 124 acquires the sensor data regarding the movement of the HMD100. The posture sensor 124 includes at least a three-axis accelerationsensor and a three-axis gyro sensor.

A communication control portion 128 transmits data output from thecontrol portion 120, to the external information processing device 10through a network adapter or an antenna by wired or wirelesscommunication. In addition, the communication control portion 128receives data from the information processing device 10, and outputs itto the control portion 120.

When the control portion 120 receives the game image data or the gamesound data from the information processing device 10, the controlportion 120 supplies the game image data to the display panel 130 andmakes the display panel 130 display the image or supplies the game sounddata to a sound output portion 132 and makes the sound output portion132 output the sound. The display panel 130 includes the display panelfor the left eye 130 a and the display panel for the right eye 130 b,and a pair of parallax images are displayed on the display panels.Further, the control portion 120 causes the sensor data from the posturesensor 124, the sound data from a microphone 126, and captured imagedata from the image sensing devices 14 to be transmitted from thecommunication control portion 128 to the information processing device10.

FIG. 4 depicts the external shape of the input device 16. FIG. 4(a)depicts the front shape of the input device 16, and FIG. 4(b) depictsthe back shape of the input device 16. The input device 16 includes acase body 20, a plurality of operation members 22 a, 22 b, 22 c, and 22d (hereinafter referred to as the “operation members 22” when they arenot particularly distinguished) operated by the user, and a plurality ofmarkers 30 a to 30 t (hereinafter referred to as the “markers 30” whenthey are not particularly distinguished) for emitting light to theoutside of the case body 20. The operation members 22 are arranged onthe head of the case body 20, and include an analog stick for performinga tilt operation, a press button, a trigger button for inputting apulled amount, and the like.

The case body 20 includes a grasping portion 21 and a curved portion 23that couples a case body head portion and a case body bottom portion,and the user passes fingers ranging from the index finger to the littlefinger between the grasping portion 21 and the curved portion 23 so asto grasp the grasping portion 21. In a state where the user grasps thegrasping portion 21, the user operates the operation members 22 a, 22 b,and 22 c with the thumb and operates the operation member 22 d with theindex finger. Although the markers 30 h, 30 i, and 30 j are provided onthe grasping portion 21, they are arranged in such positions that theyare not hidden by the hand even in a state where the user grasps thegrasping portion 21. At least one or more markers 30 are provided on thegrasping portion 21, thus enhancing the accuracy of tracking.

The marker 30 is a light emitting portion that emits light to theoutside of the case body 20, and the marker 30 may constantly emit lightwhile being tracked. The marker 30 includes a light emitting elementsuch as an LED and a resin portion that covers the light emittingelement to diffusely emit light to the outside. Note that the marker 30may be configured to emit light guided by a light guide member such as alight guide pipe to the outside. In this case, the light emittingelement such as an LED may be connected to the incident port of thelight guide member, and the marker 30 may include the emission port ofthe light guide member and a resin portion that covers the emission portto diffusely emit light to the outside.

FIG. 5 depicts a functional block of the input device 16. A controlportion 50 receives operation information that is input to the operationmembers 22, and also receives sensor data acquired with a posture sensor52. The posture sensor 52 acquires the sensor data regarding themovement of the input device 16, and includes at least a three-axisacceleration sensor and a three-axis gyro sensor. The control portion 50supplies, to a communication control portion 54, the operationinformation and the sensor data that are received. The communicationcontrol portion 54 transmits the operation information and the sensordata output from the control portion 50, to the information processingdevice 10 through a network adapter or an antenna by wired or wirelesscommunication. Further, the communication control portion 54 may acquirean instruction to emit light from the information processing device 10.The control portion 50 emits light from a plurality of markers 30, basedon the instruction to emit light supplied from the informationprocessing device 10.

FIG. 6 depicts an example of part of an image obtained by imaging theinput device 16. As illustrated in the figure, the captured imageincludes the images of the markers 30 that emit light. In the HMD 100,the communication control portion 128 transmits the image data obtainedby imaging by the image sensing devices 14 to the information processingdevice 10, and the information processing device 10 extracts the imagesof the markers 30 from the image data. Since the three-dimensional shapeof the input device 16 and the position coordinates of the plurality ofmarkers 30 arranged on the surface thereof are known, the informationprocessing device 10 solves a Perspective n-Point (PnP) problem by thedistribution of the images of the markers 30 within the imaging image soas to estimate the position and posture of the input device 16 withrespect to the image sensing devices 14.

In order for the information processing device 10 to solve the PnPproblem, it is assumed that, in whatever posture the input device 16 isimaged, the marker images are accurately identified within the capturedimage. Hence, it is necessary to appropriately design the arrangement ofthe plurality of markers 30 in the input device 16.

FIG. 7 depicts the shape of a right side surface of the input device 16.As illustrated in the figure, the curved portion 23 has a ridge surface20 c in its center, and has a first surface 20 a and a second surface 20b on both sides thereof. The ridge surface 20 c, the first surface 20 a,and the second surface 20 b are each formed as curved surfaces thatbulge outward when seen from the center portion of the input device 16,and the markers 30 c, 30 d, 30 e, and 30 f are provided on the firstsurface 20 a while the markers 30 p and 30 q are provided on the secondsurface 20 b.

FIG. 8 depicts a schematic configuration of the curved portion. An arrowA indicates the optical axis direction of the image sensing device 14.When the image sensing device 14 images the input device 16 in theposture in which the optical axis direction of the image sensing device14 is substantially parallel to the first surface 20 a, the marker 30 dand the marker 30 e provided on the first surface 20 a are imaged in astate of being close to each other. Since the image sensing device 14images the side of the first surface 20 a but not the side of the secondsurface 20 b, the marker 30 p is not imaged.

At this time, since the information processing device 10 cannotdistinguish between the two marker images within the captured image, theinformation processing device 10 may erroneously recognize the twomarker images as one marker image. In such a case, the informationprocessing device 10 cannot correctly solve the PnP problem, and thusthe information processing device 10 fails in the estimation of theposition and posture of the input device 16.

The present inventors have conducted various simulations and experimentsto find that this failure was caused by the fact that the image of themarker 30 d and the image of the marker 30 e were excessively close toeach other within the imaging image and to obtain such findings thatincreasing a distance between the marker 30 d and the marker 30 e on thefirst surface 20 a makes it possible to separate the positions of thetwo marker images within the captured image. From the findings describedabove, the present inventors have focused on the normal directions ofthe markers 30 and found that, when an angle difference between thenormal directions of the two markers 30 is small, the two marker imagesmay be close to each other within the captured image in a specificposture.

Here, the normal direction of the marker 30 is defined as a directionperpendicular to the surface of the case body 20 on which the marker 30is provided. For example, when the marker 30 is configured as an LEDchip that is arranged parallel to the surface of the case body 20, thenormal direction of the marker 30 may be defined as the optical axiscenter direction of the LED chip. Further, when the marker 30 includes aresin portion for diffuse emission in the emission port of a light guidepipe, the normal direction of the marker 30 may be defined as a lineperpendicular to the contact surface of the center of a diffuse lightemission surface.

Hence, in the input device 16 of the embodiment, when an angledifference between the normal directions of two markers 30 is less thana predetermined threshold value Ath, a distance (linear distance)between the two markers 30 is equal to or greater than a predeterminedfirst distance D1. Note that the values of the threshold value Ath andthe first distance D1 are determined by various types of conditions suchas the resolution of the image sensing device 14. For example, undercertain conditions, the threshold value Ath may be 35 degrees, and thefirst distance D1 may be 3 cm. When, in an example illustrated in FIG.8, the angle difference between the normal directions of the marker 30 dand the marker 30 e is less than the threshold value Ath, the distancebetween the marker 30 d and the marker 30 e is equal to or greater thanthe first distance D1. In this way, even when the input device 16 isimaged in the direction of the arrow A, the information processingdevice 10 can recognize each of the two marker images within thecaptured image.

The present inventors have also obtained such findings that, since, inthe imaging in the direction of the arrow A, the marker 30 p on thesecond surface 20 b is not imaged in the first place, a distance betweenthe marker 30 p and the marker 30 d or the marker 30 e on the firstsurface 20 a is allowed to be shorter than the first distance D1. Thepresent inventors found, from the findings described above, that, whenan angle difference between the normal directions of two markers 30 islarge, the two marker images are prevented from being (or are unlikelyto be) close to each other within the captured image.

FIG. 9 is a diagram for illustrating the normal directions of twomarkers 30 on the same surface. Here, illustrated is an example where amarker 30 v and a marker 30 w are provided on the same surface that iscurved in a protruding shape with respect to the direction of imaging bythe image sensing device 14. At this time, when the image sensing device14 images the surface in a direction indicated by an arrow B, the marker30 v is imaged but not the marker 30 w. This is caused by the fact that,since the surface where the markers are provided is curved in aprotruding shape, the marker 30 w on the back side with respect to theimage sensing device 14 is hidden from the image sensing device 14 by asurface bulging between the marker 30 v and the marker 30 w.

Hence, when the angle difference between the normal directions of thetwo markers 30 on the same surface is large, even if the surface isimaged in the direction of the arrow B, the marker 30 v is not imaged,with the result that the images of the two markers 30 are prevented frombeing close to each other within the captured image. Note that, forreference, when the image sensing device 14 shots the surface in adirection indicated by an arrow C, the image of the marker 30 v and theimage of the marker 30 w are originally present in positions away fromeach other within the captured image, and thus the informationprocessing device 10 can distinguish between the two markers withoutfail, with the result that the two marker images are prevented fromerroneously being recognized as one marker image.

Therefore, in the input device 16, when an angle difference between thenormal directions of two markers 30 is equal to or greater than thepredetermined threshold value Ath, a distance (linear distance) betweenthe two markers 30 is equal to or greater than a predetermined seconddistance D2. Note that, the second distance D2 is shorter than the firstdistance D1. The value of the second distance D2 is also determined byvarious types of conditions such as the resolution of the image sensingdevice 14, and for example, even when the distance between the imagesensing device 14 and the input device 16 is the largest, the value ofthe second distance D2 is designed to be such a length that the twomarker images can be distinguished within the captured image. Forexample, under certain conditions, the second distance D2 may be 2 cm.When, in the example illustrated in FIG. 8, the angle difference betweenthe normal directions of the marker 30 d and the marker 30 p is equal toor greater than the threshold value Ath, the distance between the marker30 d and the marker 30 p is equal to or greater than the second distanceD2.

As described above, in the input device 16 of the embodiment, a distancebetween two markers 30 is designed based on an angle difference betweenthe normal directions of the markers 30. In the design of thearrangement of the markers 30, the markers 30 are randomly arranged in aplurality of positions of the input device 16 whose three-dimensionalshape is known, and whether such an arrangement is the optimalarrangement is checked by simulation or the like. Here, such conditionsas the threshold value Ath described above are used in the simulation orthe like, and thus optimal arrangement of the markers 30 can berealized.

The present invention has been described above based on the embodiment.The embodiment described above is illustrative, and a person skilled inthe art would understand that various variations of the constituentelements thereof and the combination of processing processes arepossible and that the variations are also within the scope of thepresent invention.

Although, in the embodiment, the arrangement of a plurality of markers30 in the input device 16 including the operation members 22 isdescribed, the device to be tracked does not necessarily need to includethe operation members 22. In addition, although, in the embodiment, theimage sensing devices 14 are attached to the HMD 100, the image sensingdevices 14 may be attached to positions other than the HMD 100 as longas the image sensing devices 14 can image marker images.

INDUSTRIAL APPLICABILITY

The present invention relates to a device including a plurality ofmarkers.

REFERENCE SIGNS LIST

-   1: Information processing system-   10: Information processing device-   14: Image sensing device-   16: Input device-   20: Case body-   21: Grasping portion-   22: Operation member-   23: Curved portion-   30: Marker

1. A device comprising: a case body; and a plurality of markersconfigured to emit light to an outside of the case body, wherein, whenan angle difference between normal directions of two markers is lessthan a predetermined threshold value, a distance between the two markersis equal to or greater than a predetermined first distance.
 2. Thedevice according to claim 1, wherein, when an angle difference betweennormal directions of two markers is equal to or greater than apredetermined threshold value, a distance between the two markers isequal to or greater than a predetermined second distance, and the seconddistance is shorter than the first distance.
 3. The device according toclaim 1, wherein a distance between two markers is designed based on anangle difference between normal directions of the markers.
 4. The deviceaccording to claim 1, wherein the device is an input device including anoperation member operated by a user.
 5. The device according to claim 1,wherein the case body includes a grasping portion grasped by the user,and one or more of the markers are arranged on the grasping portion. 6.The device according to claim 1, wherein the device is imaged with animage sensing device attached to a head-mounted display.